Low light image enhancement using curvelet transform and iterative back projection

With the advancement of technology in image capturing, people are accustomed to high-resolution images. One of the primary necessities of an image capturing system is to provide the same. However, in many cases, the image resolution may not be reaching the expectations of the user which leads to a decrease in user experience. This is a common phenomenon that occurs when the images are captured in low light or if the image encounters a distortion either because of lack of exposure or the image capturing devices may be equipped with a small size sensor. In this work, a resolution enhancement technique using the concepts of curvelet transform and iterative back projection is presented. Sparse representation of images can be enhanced using a combination of curvelet transforms with iterative back projection. Application of curvelet transform along with iterative back projection algorithm on low light images results in enhancing the resolution of the images. The resultant images from here then passed through the inverse transform block and gives an image with contrast enhancement which leads to the user experience improvement. The antiquated image enhancement with improvement in the resolution is validated with the measurement of peak signal-to-noise ratio and structural similarity index. The usage of curvelet transform with iterative back projection leads to the restoration of the image resolution by minimizing the distortions, thus leading to an enhanced image whose edge details are retained.

With the advancement of technology in image capturing, people are accustomed to high-resolution images. One of the primary necessities of an image capturing system is to provide the same. However, in many cases, the image resolution may not be reaching the expectations of the user which leads to a decrease in user experience. This is a common phenomenon that occurs when the images are captured in low light or if the image encounters a distortion either because of lack of exposure or the image capturing devices may be equipped with a small size sensor. In this work, a resolution enhancement technique using the concepts of curvelet transform and iterative back projection is presented. Sparse representation of images can be enhanced using a combination of curvelet transforms with iterative back projection. Application of curvelet transform along with iterative back projection algorithm on low light images results in enhancing the resolution of the images. The resultant images from here then passed through the inverse transform block and gives an image with contrast enhancement which leads to the user experience improvement. The antiquated image enhancement with improvement in the resolution is validated with the measurement of peak signal-to-noise ratio and structural similarity index. The usage of curvelet transform with iterative back projection leads to the restoration of the image resolution by minimizing the distortions, thus leading to an enhanced image whose edge details are retained.
Resolution enhancement of images is a key requirement especially when the images are captured in an environment where the light present is of considerably low. Such images which we call low light images are often subjected to a lot of degradation in terms of noise and resolution and thus a mechanism to improve the same is of utmost concern. In such cases, as an addition to resolution enhancement, an improvement over the contrast by maintaining the edge details is considered to be a value addition to the image processing techniques to be applied. High resolution images find their applications spread over a number of varied areas but are not limited to the field of medical imaging, remote sensing, satellite imaging for the classification of urban land areas, and other commercial applications. When such images are combined with enhanced contrast, the tremendous potential of its applicability encompasses even wider regions. This creates the potential for this research work which concentrates on tapping the same into more meaningful image processing techniques.
With the tremendous advancements in technology especially related to image capturing and video capturing, there has been a multi-fold increase in the number of applications making use of the same. The image capturing or video capturing systems are expected to operate not only in presence of appreciable light but also in low light, especially during the night times. With the application extending to such cases, the low light images that are captured by the systems are not clear enough to serve the purpose of what they are supposed to serve. The images which are captured in low light are susceptible to a decrease in contrast as well as a reduction in its resolution. The scene details may also appear to be blurred with a reduction in the image resolution. This creates a serious concern especially when the image capturing or video capturing systems are being used for security surveillance. The application areas where low light images are usually appearing include medical imaging, which again is a critical application where the image resolution is of prime importance. Since the application areas involve such serious use cases, low light image enhancement finds its application more as a necessity rather than a mere feature.
The low light images with reduced contrast and resolution become one of the major concerns in the applications which uses them. Thus resolution enhancement of low light images coupled with contrast enhancement will result in providing a reasonable solution to the problem at hand. Image enhancement using a combination of Iterative back projection and Curvelet transforms is proposed in this research work for resolution and contrast www.nature.com/scientificreports/ enhancement. Iterative back projection is a principal concept in image reconstruction. This concept can be visualized as a sketch artist who sketches a portrait of a person with the help of certain key features. Even though the artist may not be able to completely replicate the exact portrait, however, the artist may be successful in drawing a reasonably good portrait by drawing the features explained. The more the number of features, the closer will be the drawn product to the actual requirement. Several enhancements have been made to this fundamental concept of iterative back projection, but still, it has its use in image reconstruction. The rest of the paper is organized as follows: Section "Literature review" gives the brief about the research articles referred, Section "Methodology" is the description of the proposed research work, Section "Simulation results" is results and comparison using performance metrics and Section "Conclusion" is the conclusion and scope for future work.

Literature review
Low light image restoration or enhancement is a technique where a digital image is improved upon its quality so that it becomes feasible to use them for applications where quality plays a crucial role. A considerable amount of literature related to image enhancement has been reviewed. Through this review, it was understood that the enhancement algorithms are classified into those working in the spatial domain and those in the transform domain. The enhancement can also be performed using several filtering techniques. The transform domain techniques are again classified into wavelet domain and curvelet domain algorithms based on the domain in which they work. The reviewed papers include resolution enhancement with iterative back projection 1-4 , the papers which explain the concept of Gaussian Mixture Model(GMM) and curvelet transform 5-7 .

Methodology
In this section, the method proposed in this work is presented. The block diagram of the entire process is shown in the Fig. 1. In the proposed method, a low light, low resolution image is taken as the input image. Input is then transformed to the curvelet domain. Curvelet transform gives good sparsity for images with curves and edges. The low light images, in general, contain a lot of noise, and in order to remove the noise a pre-filtering stage is applied. This pre-filtering reduces the image quality in terms of resolution and in order to improve the image resolution, an iterative back projection algorithm with gradient descent optimization is used here. Usage of this modified method which uses iterative back projection with curvelet transform not only improves the resolution of the image but also retains the edge details from the image. Iterative back projection (IBP) is a process that initially performs an approximate guess and progresses iteratively to minimize the error that is observed between the estimated image and the input image. Once the error is minimized to an acceptable level, the iterations are stopped. The output obtained at this stage is a resolution enhanced image. This, in turn, passes through the inverse curvelet stage and the final output will be an enhanced quality image both in terms of contrast and resolution. The details of the different stages involved in the process are given below: Stage 1 This stage is a curvelet decomposition stage, where the image is decomposed into different resolution layers, The details of the steps involved in curvelet transform implementation are given below: www.nature.com/scientificreports/ • Sub-band Decomposition It divides the input into different frequency layers. This operation is more similar to a convolution operation. Decomposed layers are given as where I is the input to be decomposed, P 0 is the low frequency layer and � 1 , � 2 , . . . are the high frequency layers.

• Smooth Partitioning
It performs a scaling operation where each frequency layer is divided into squares of appropriate scale. A grid of dyadic squares, Q is defined for this purpose. It is represented as Q represents the group of all Q and Q s is the collection of dyadic squares of scale s. w is the windowing function used for smooth partitioning and displacement of window localized near Q is represented as w Q . Dividing the sub-band into squares is done by multiplying s I with w Q .

• Renormalization
It is the process of normalizing the dyadic squares into unit squares. For each Q the transport operator S Q on I is defined as and normalizing the squares are done as

• Ridgelet Analysis
Ridgelet transform is used to analyze the normalized squares. In this step, the ridgelet transform is applied to the combined dyadic squares.
Stage 2 This stage is a pre-filtering process and this pre-filtering technique uses a Gaussian Mixture Model based denoising process 5 . Stage 3 This stage consists of the application of an iterative back projection algorithm to the coarse component of the transformed image. Here the IBP algorithm uses a gradient descent optimization technique which helps in achieving high quality solutions. The optimization technique uses a constant step function in order to optimize the selection of HR images to obtain the estimation of the same. The main objective of IBP is to minimize the error between the simulated and observed data. The steps are given as follows: Step 1:Make the initial guess of the high resolution image by decimating the low resolution input image.
Step 2:Using this high resolution image simulate a low resolution version of it with the help of a down sampling operation.
where I CLR is the coarse component of the low resolution image, I CHR is the coarse component of the high resolution image, ↓ d is the decimation operator and B is the degradation function. (n) represents n th iteration.
Step 3:Estimate the HR image by back propagating the difference between the input low resolution image and simulated low resolution image.
H BP is the back projection kernel.
Step 4:Check the error, which is the difference between the input low resolution image and simulated low resolution image.
Step 5:Continue steps 2, 3 and 4 until the energy of the error is minimum.The output of this stage is high resolutiontion coarse component.
Step 4 This is the final stage in the enhancement process. In this stage, the inverse curvelet transform of the resolution enhanced coarse component is executed. The different steps involved in this stage are Ridgelet Synthesis, Renormalization, Smooth integration, and Sub-band recomposition. This is in essence is the reverse of the process detailed in Stage 1. The final output after the inverse curvelet transform is an image with enhanced quality both in terms of resolution and contrast.
The analysis of the quality of the enhanced image is done with the help of the quality metrics, Peak Signal to Noise Ratio(PSNR) and Structural Similarity Index(SSIM) 8 .

Simulation results
Simulations were done in MATLAB2018a and a set of natural low light images were used to analyse the performance of the proposed method. A database of low light images was built for experimental purposes. This database which is known as 'Dimair' 9 has approximately 300 images acquired in a low light environment and used as input to the proposed research work. The resolution enhancement results from the proposed method were compared with the results of resolution enhancement obtained using conventional iterative back projection algorithm 2 and resolution enhancement method which uses Edge Preserving IBP 3 . Images of Car, Sculpture, Building, and Sky from the 'Dimair' data set are taken for the simulation. The images selected for simulation were all captured in low light environment and the intensity of light was measured using a lux meter. These meter readings were taken at the instants when the images were captured. The simulation was performed on images of car which are taken at light intensities 232,262,282,315, images of sculpture captured at light intensities 84,112,140,152, images of the building taken at light intensities 230,253,298,321, and images of sky captured at light intensities 270,287,302,330 (unit is lux). For all these images, during a preprocessing stage 9 , the image quality was reduced which resulted in deterioration of resolution. The proposed method was applied to this low resolution images along with conventional iterative back projection resolution enhancement algorithm 2 and resolution enhancement method which uses Edge Preserving IBP 3 . Figures 2, 3    www.nature.com/scientificreports/ shown, it can be observed that the proposed method gives better visual quality for images compared to the conventional iterative back projection method and resolution enhancement method which uses Edge Preserving IBP. For all the images at different intensity levels, PSNR and SSIM values were calculated and noted. In the above figures, PSNR and SSIM values for that particular intensity level are mentioned below each image. It is seen that the proposed method is better in terms of both PSNR and SSIM.
Also the PSNR and SSIM values are tabulated for the images with different intensity levels and are shown in Tables 1 and 2. It is clear from the tables that the proposed method gives better results in terms of PSNR and SSIM values compared to the conventional IBP algorithm and resolution enhancement method which uses Edge Preserving IBP. The tabulated PSNR and SSIM values for different images are plotted against the intensity level of the images. Figures 6, 7, 8, and 9 show the graph of PSNR and SSIM plotted against light intensities for Car, Sculpture, Building and Sky images. From the graph, we can see that the proposed method is efficient in enhancing the resolution of degraded images at different intensity levels.
The result of contrast enhancement on the low light images for one intensity level is given in Fig. 10 along with the original images. The Car image is at an intensity level of 315 lux, the Sculpture image at 152 lux, the Building image at 321 lux, and the Sky image at 330 lux. The histogram plot for these original images and the contrast enhanced images are shown in Fig. 11. The difference in contrast between the original image and contrast enhanced image is clearly visible from the plotted histogram.
In order to check the improvement in terms of contrast, we have calculated Absolute mean brightness difference and entropy for the ground truth and contrast enhanced image from the method which combines curvelet and IBP. The values of mean brightness and absolute brightness difference is recorded in the Table 3. The values of entropy for the captured image and contrast enhanced image is tabulated in the Table 4. From Table 3, it is seen that the mean brightness of the contrast enhanced output image is higher than the mean brightness of the input ground truth images taken for simulation. This clearly indicate there is an improvement in the contrast of the output image. The value of difference in brightness is also recorded in the same table. From the entropy values of the input ground truth and entropy values of the enhanced image given in Table 4, we can see that the  www.nature.com/scientificreports/ entropy of the enhanced image is higher than the entropy of the input image. This shows the efficiency of the proposed algorithm in retaining the information content.

Conclusion
An image enhancement method that combines curvelet transform and iterative back projection is implemented in this paper. The quality of the output image is improved in terms of both resolution and contrast. A database 'Dimair' has been built to store low light images and the experiment was done by taking a few of these images as www.nature.com/scientificreports/ the input. The traditional iterative back projection and the method which uses edge preserving IBP are compared with the method that combines curvelet transform with iterative back projection, in terms of PSNR and SSIM, and the results suggest affirmative for the proposed method. The proposed method has been found to perform better not only with the enhancement of resolution but also with the performance in terms of edge enhancement with reduced ringing artefacts. Here the primary aim was to improve the image resolution and the work can be further extended by introducing techniques to have more improvements in contrast enhancement of colour images acquired in the low light environment.   www.nature.com/scientificreports/

Data availability
All data generated or analysed during this study are included in this published article. Also they are available from the corresponding author upon request.